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1. 


Introduction 


The  author  [9)  previously  developed  and  evaluated  some  heuristic 
procedures  for  seeking  a  good  approximate  solution  of  any  pure  integer 
linear  programming  problem^ 


x.  is  a  nonnegative  integer  (j  =  1,2,  ...  ,  n)  , 

J  1 

\  l 

having  no  implicit  or  explicit  equality  constraints."  It  was  found  that 
the  procedures  are  extremely  efficient,  being  computationally  feasible 
for  problems  having  hundreds  of  variables  and  constraints.  Furthermore, 
they  proved  to  be  very  effective  in  identifying  good  solutions,  often 
obtaining  optimal  ones.  Thus,  the  procedures  provide  a  way  of  dealing 
with  the  frequently  encountered  integer  programming  problems  that  are 
beyond  the  computational  capability  of  existing  algorithms.  For  smaller 
problems,  they  also  provide  an  advanced  start  for  accelerating  certain 
primal  algorithms,  including  the  author’s  Bound-and-Scan  algorithm 
and  Faaland  and  Hillier's  Accelerated  Bound-and-Scan  algorithm.f-4}-;*^ 

In  addition,  Jeroslow  and  Smith  have  found  that  imbedding  the  first  part 

of  one  of  these  procedures  inside  the  iterative  step  of  a  branch-and-bound 
algorithm  can  greatly  improve  the  latter's  efficiency  in  locating  solutions 
whose  objective  function  value  is  within  a  specified  percentage  of  that  for 
the  optimal  solution. 


All  of  the  procedures  use  the  same  general  three-phase  approach, 
which  can  be  described  conceptually  as  follows.  Phase  1  identifies  a 
general  region  within  which  to  explore  for  good  feasible  solutions  by 
finding  (l)  the  optimal  noninteger  solution  by  the  simplex  method,  and 
(2)  a  second  point  well  into  (or  beyond)  the  feasible  region.  Phase  2 
then  conducts  this  exploration  by  slowly  moving  along  the  line  segment 
from  this  first  point  to  the  second  while  searching  nearby  for  a  feasible 
(integer)  solution.  Phase  3  attempts  to  move  from  the  feasible  solution 
obtained  to  a  succession  of  better  ones.  The  final  solution  obtained 
is  the  desired  approximate  solution.  If  it  is  crucial  to  increase  the 
probability  of  obtaining  an  optimal  solution,  one  can  continue  by  identi¬ 
fying  many  good  feasible  solutions  in  Phase  2  and  then  applying  Phase  3  to 
each  of  them,  thereby  yielding  multiple  final  solutions  from  which  to  choose. 

The  previous  paper  [9]  presents  alternative  methods  for  conducting  each 
of  these  phases,  thereby  yielding  36  distinct  overall  procedures.  (These 
procedures  are  labeled  x-y-z  to  indicate  that  Methods  x,  y  and  z  are  to 
be  used  in  Phases  1,  2  and  3,  respectively.)  A  program  of  computational  exper¬ 
imentation  identified  four  types  of  procedures  (1-2-1,  2-2-1,  1-3-1,  and  2-3-1) 
that  appear  to  be  substantially  better  than  the  others,  but  this  experimentation 
was  unsuccessful  in  detecting  significant  differences  among  the  four.  Further¬ 
more,  only  tentative  conclusions  can  be  drawn  in  comparing  the  four  alternative 
criteria  (A,  B,  C,  and  D)  for  conducting  a  certain  test  in  Phase  2.  Thus,  16 
distinct  procedures  (l-2i-l,  2-21-1,  1-31-1,  and  2-31-1  for  i  =  A,B,C,D) 

still  await  definitive  comparison.  Another  question  of  this  study  which  was 
only  partially  answered  concerns  the  best  way  in  which  to  generate  multiple 
solutions . 

The  present  paper  has  two  main  purposes.  One  is  to  briefly  present 
some  promising  new  methods  for  conducting  each  of  the  three  phases. 


This  is  done  in  the  next  three  sections.  The  second  objective  is  to 

address  some  unanswered  questions  mentioned  above  (and  outlined  in  more 
detail  in  the  last  section  of  [9  ))  in  the  broader  context  of  these  new 
procedures.  Thus,  a  comprehensive  testing  program  has  been  conducted 
to  further  evaluate  and  compare  the  best  of  the  old  procedures  with  the 
new  ones  presented  here.  Procedures  for  generating  multiple  solutions  also 
are  discussed  and  tested.  The  test  results  and  conclusions  are  presented 
in  the  final  three  sections  and  the  appendix. 

Various  other  investigations  also  have  proposed  heuristic  algorithms 
for  integer  programming  in  recent  years.  These  include  Reiter  and  Rice  [14], 
F.chols  and  Cooper  [2],  Senju  and  Toyoda  [16],  Roth  [15],  Kochenberger ,  McCarl 
and  Wyman  [13], Toyoda  [17],  Balas  and  Martin  [1],  and  Glover  [5].  Also  of 
particular  interest  here  is  the  heuristic  algorithm  of  Ibaraki,  Ohashi,  and 
Mine  [10],  which  extends  (with  some  modification)  the  author's  original 
heuristic  procedures  [9]  to  mixed  integer  programming.  (See  Section  9  for  a 
comparison  of  this  algorithm  with  the  procedures  proposed  here.)  In  addition, 
Faaland  and  Hillier  [3]  have  extended  the  analysis  and  development  of  Phase  1 
methods  considerably  beyond  the  present  paper. 

For  the  sake  of  brevity,  the  presentation  here  will  not  repeat  most 
of  the  relevant  material  from  [9],  so  the  reader  is  advised  to  first  read 
this  earlier  paper  and  to  keep  it  available  for  reference  purposes  as  he 
proceeds  through  the  following. 

2.  A  Multiple  Solution  Approach  to  Phase  1 

It  would  sometimes  be  worthwhile  to  take  the  time  to  generate  multiple 
final  solutions  in  order  to  try  to  improve  upon  the  initial  one  obtained  after 
completing  all  three  phases  once.  As  mentioned  above,  one  method  of  doing  this, 
based  on  repeated  applications  of  Phase  2,  is  presented  in  [9].  Another  approach 
described  below  is  based  instead  on  repeated  applications  of  Phase  1. 
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This  approach  involves  generating  a  sequence  of  distinct  pairs  of 

points,  x^  and  x^^.  Recall  that  the  original  x^^  is  just  an 

optimal  solution  to  the  problem  without  the  integer  restriction,  as 

obtained  by  the  simplex  method.  Each  new  x^  is  then  obtained  directly 

from  this  one  by  choosing  an  adjacent  extreme  point  in  the  polyhedral  set 

of  feasible  solutions  for  this  linear  programming  problem.  This  adjacent 

extreme  point  is  obtained  simply  by  performing  a  single  pivot  from  the 

optimal  basis.  The  method  used  here  for  choosing  the  sequence  of  adjacent 
extreme  points  is  based  on  the  size  of  the  simplex  multipliers  (i.e.,  reduced  costs) 

Thus,  the  first  pivot  is  performed  on  the  nonbasic  variable  having  the 

smallest  (negative)  simplex  multiplier,  the  second  on  the  one  having  the 

second  smallest  multiplier,  etc.  This  process  was  stopped  after  obtaining 

a  preassigned  number  of  new  values  of  x^,  which  was  taken  to  be  five 

for  the  computational  experimentation.  (Note  that  another  alternative  would 

be  to  examine  all  of  the  adjacent  extreme  points  and  then  choose  the  five 

best  ones  according  to  their  objective  function  value.) 

For  each  x^  ,  the  chosen  Phase  1  method  would  be  used  to  obtain 

(2) 

the  corresponding  Value  of  x  in  the  usual  way.  Connecting  these 

two  points  provides  a  line  segment  wdiich  is  completely  distinct  from 

the  original  one,  but  which  still  passes  through  a  promising  region  for 

searching  for  a  good  feasible  solution.  Therefore,  Phase  2  and  then 

Phase  3  would  be  applied  just  as  before  with  respect  to  the  new  x^ 

(2) 

and  x  in  order  to  obtain  the  new  final  solution.  Doing  this  in 
turn  with  each  new  pair  of  such  points  provides  the  desired  multiple  solu¬ 
tions,  from  which  the  best  one  would  be  chosen  as  the  solution  to  use. 

A  very  desirable  characteristic  of  any  method  of  generating  multiple 
solutions  would  be  that  each  new  solution  generated  have  a  relatively 


high  probability  of  being  distinct  from  the  preceding  ones  while  still 
tending  to  have  an  objective  function  value  that  is  at  least  close  to  and 
sometimes  is  better  than  that  for  the  initial  final  solution.  The  fact  that 
the  original  method  frequently  merely  regenerated  a  previous  final  solution 

was  a  major  motivation  for  developing  the  new  approach  described  above. 

It  was  anticipated  that  having  the  Phase  2  search  proceed  from  a  distinct 
new  line  segment  rather  than  merely  moving  further  down  the  same  segment 
might  increase  the  chances  of  obtaining  a  distinct  final  solution  that 
was  still  a  good  one. 

This  new  approach  will  be  designated  by  inserting  an  R  (for  "Repeat") 
before  the  number  of  the  method  being  used  for  each  application  of  Phase  1. 
Similarly,  the  original  approach  wi31  be  designated  by  inserting  an  R 
before  the  number  of  the  method  being  used  for  each  application  of  Phase  2. 
For  example,  the  labeling  for  using  procedure  1-HA-l  to  generate  multiple 
solutions  by  the  new  and  original  approaches  would  be  R1-2A-1  and  1-R2A-1, 
respectively. 

1 .  New  Criteria  in  Phase  2 

Recall  that  Methods  2  and  3  of  Phase  2  involve  iteratively  moving 
from  one  infeasible  integer  solution  to  another  which  is  "less  infeasible." 
in  a  certain  sense  (or,  if  such  an  improvement  is  not  possible,  beginning 
another  cycle  of  such  iterations  with  a  new  starting  solution).  Each 
such  move  involves  changing  one  variable  by  +1.  Four  different  criteria 
for  choosing  the  variable  to  be  changed  were  presented.  Criteria  A  and 
3  focus  exclusively  on  the  constraints,  and  thereby  would  seem  to  run 
some  risk  of  leading  to  a  feasible  solution  with  a  relatively  poor  value 
of  the  objective  function.  On  the  other  hand,  criteria  C  and  D  give 
'.onsiderable  weight  to  the  objective  function,  but  in  a  way  that  sometimes 


5 


gives  preference  to  moves  that  point  above  rather  than  toward  the  feasible 
region,  which  would  seem  to  increase  the  risk  of  overlooking  feasible 
solutions  subsequently.  This  analysis  would  suggest  that  a  compromise 
between  these  two  types  of  criteria  might  be  appropriate.  One  such  com¬ 
promise  is  described  below. 


Using  the  notation  and  terminology  of  [9 ],  the  new  criterion  defines 


the  "improvement"  from  changing  the  value  of  a  variable  x.  as 

1 


p  =  -Aq  +  c*  Ax  , 

«J  J 


where  c!  is  the  normalized  value  of  c.  and  Ax.  is  the  change  in 
J  3  1 

x..  Thus,  this  is  just  the  definition  used  for  criteria  A  and  B  (p  =  -Aq) 

3 

except  for  adding  a  term  reflecting  the  effect  of  the  change  on  the  objective 

function.  In  the  case  of  criterion  A,  where  the  first  definition  of  q 

is  used,  this  added  term  has  a  very  natural  interpretation.  In  particular, 

suppose  that  a  lower  bound  bQ  on  an  acceptable  value  of  the  objective 

function  is  introduced  explicitly  as  a  constraint,  cx  >  bQ,  and  that 

b  exceeds  the  value  x  attained  by  the  current  solution  both  before 
o  o 

and  after  the  change  in  x  ^ .  Then  the  resulting  criterion  A  definition 
of  p  coincides  exactly  with  the  new  definition  given  above.  Thus, 
in- effect,  the  new  criterion  encourages  large  moves  toward  the  feasible 
region  (as  with  criteria  A  and  B)  but  with  a  modified  interpretation 
of  feasibility  that  particularly  encourages  movement  toward  the  most 
attractive  portion  of  the  feasible  region  (as  with  criteria  C  and  D). 

The  mechanics  of  applying  the  new  criterion  are  to  proceed  through 


Phase  2  exactly  as  if  criterion  A  or  B  were  being  used  (depending  on 
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which  definition  of  q 


is  adopted)  except  when  there  are  no  q*  <  0 

J  ’ 

and  part  (c)  of  Step  8  is  entered,  in  which  care  merely  substitute  the 

new  definition  of  p  for  each  p.  to  ce  calculated.  Since  the  results  of 

d 

computational  testing  distinctly  favor  criterion  A  over  criterion  B  (see 
Sections  5  and  6),  only  the  first  definition  of  q  (the  one  used  by  criterion 
A)  was  used  with  the  new  criterion,  and  this  version  will  be  called  criterion  E. 

It  should  be  noted  that  there  also  are  variations  of  this  criterion 

that  conceivably  could  be  slightly  better.  For  example,  a  factor  other 

than  one  could  be  used  for  the  new'  term  in  order  to  give  a  different 

relative  weighting  between  the  constraints  and  objective  function,  or 

the  new  term  could  be  deleted  when  it  is  positive  (effectively  setting 

b  eoual  to  x  before  changing  a  variable),  etc. 
o  *  o 

Another  viewpoint  is  that  it  is  really  combinations  of  moves  that 
are  particularly  important  in  moving  expeditiously  to  a  good  feasible 
solution.  Therefore,  any  move  that  yields  an  improvement  should  be  taken 
immediately  rather  than  wasting  time  by  identifying  and  comparing  all 
possible  improving  moves  for  this  iteration.  Other  such  moves  that  are 
truly  worthwhile  should  still  be  available  for  subsequent  iterations. 
Furthermore,  when  the  number  of  variables  is  large,  this  approach  may 
greatly  reduce  the  time  required  to  execute  Phase  2  without  significantly 
sacrificing  effectiveness. 

When  the  first  definition  of  q  is  used,  this  streamlined  approach 

will  be  designated  as  criterion  S.  It  is  applied  by  executing  Phase  2 

essentially  as  with  criterion  A  (or  c)  except  that  Steps  7  and  8  are 

bypassed.  Instead,  as  soon  as  a  a*  such  that  qt  <  q  is  found  in 

J  J 

Step  6,  the  iteration  is  terminated  immediately  by  setting  k  equal  to 


this  j  and  going  to  Step  9.  On  the  next  iteration,  Step  6  would  resume 


from  where  it  had  left  off,  calculating  (as  necessary)  q*  ,  q*  „,..., q*, 

K+ii  n 

q*,...,q*  in  this  order.  If  there  are  no  a*  such  that  q*  <  q  on  a 
*1  K  J  J 

given  iteration,  then  one  goes  to  Step  10. 


U .  New  Methods  for  Phase  3 

A  drawback  of  the  current  methods  for  Phase  3  is  that,  at  each  iter¬ 
ation,  one  attempts  to  identify  a  better  (feasible)  solution  only  by 
considering  certain  ways  of  changing  either  one  or  two  variables  in  the 
current  solution.  It  sometimes  is  necessary  to  change  many  variables  in 
order  to  reach  a  better  solution.  However,  it  clearly  would  be.  compu¬ 
tationally  infeasible  for  problems  of  significant  size  to  systematically 
consider  all  ways  of  changing  several  variables  simultaneously.  Therefore 
what  is  needed  are  methods  that  will  efficiently  consider  only  promising 
ways  of  changing  many  variables. 

As  suggested  by  Ibaraki  et  al  [10],  one  approach  of  this  kind  would 
be  to  use  a  search  similar  to  that  employed  in  phase  2.  Pecall  that 
the  Phase  2  search  allows  making  many  promising  variable  changes  in  suc¬ 
cession  in  an  attempt  to  eventually  reach  the  solution  of  interest,  namely 
a  good  feasible  solution.  Essentially  the  same  method  also  could  be  used 
in  Phase  3  in  an  attempt  to  eventually  reach  the  solution  of  interest 
there,  namely,  a  better  feasible  solution  than  the  best  one  found  thus 
far.  Thus,  one  would  deliberately  move  from  the  current  best  feasible 
solution  out  of  the  feasible  region,  and  then  try  to  move  through  a  suc¬ 
cession  of  infeasible  (integer)  solutions  that  seem  to  be  progressing 
toward  a  better  feasible  solution  (if  one  exists). 


> 


n-.-w  methods  for  Phase  3  (Kethodr  3,  !l ,  ana  5)  that  are  based 


r :  n  *hi:-  approach  are  presented  tele-:.  Given  the  current  test 

it'-  volution  x^L'  and  its  objective  function  value  =  cx^\ 


•.t.r methods  initiate  the  search  procedure  mentioned  above  by  intro- 
.'nstraint,  cx  >  b  ,  where  b 


vu  'i’o;  a  r.ew  cor 


—  o 


(L)  */ 

xy  '  +  1.-'  This  has 
o  o 


the  effect  of  making  infeasible  and  reducing  the  feasible  region 
so  that  it  includes  only  better  feasible  solutions  (if  any).  Thus,  the 
goal  becomes  to  reach  some  integer  solution  in  this  reduced  feasible 


region. 


Methods  3  and  4  go  through  n  cycles  to  search  for  a  better  solution. 

Each  cycle  begins  by  changing  one  variable  (call  it  x, )  in  the  solution 
by  either  plus  one  (if  c  >  0)  or  minus  one  (if  c,  <  0).  Thus, 
the  n  cycles  correspond  to  setting  k  =  1,  2,  ...  ,  n  in  turn.  This 
first  seep  in  each  cycle  has  the  effect  of  giving  a  new  solution  which 
usually  is  substantially  further  away  from  the  reduced  feasible  region. 

The  remaining  part  of  the  cycle  then  consists  of  essentially  applying 
Steps  4  through  o  of  Phase  2  with  this  new  solution  as  the  starting  point. 

(Criterion  A  always  was  used  for  doing  this  in  the  computational  testing  since 
criteria  E  and  S  had  not  yet  been  developed  at  this  stage  of  the  testing  program.) 
The  only  changes  that  need  to  be  made  in  Steps  4-9  are  the  following. 

First,  the  new  objective  function  constraint  is  treated  just  like  the 
other  functional  constraints  (after  making  the  obvious  conversions  in  format 
and  notation),  so  i  effectively  runs  from  0  to  m.  Second,  additional 
changes  in  x,  are  not  allowed  so  j  /  k  throughout.  Third,  having 
Q  =  c?  in  Step  3  terminates  the  cycle,  in  which  case  reset  k  =  k  +  1 


Thi  s  a 


sumes  that  all  of  the  c.  are  integers;  otherwise, 

J 

where  e  is  an  extremely  small  positive  constant. 


set 


b  = 
o 
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r 


and  start  the  next  cycle  (if  k  <  n). 

The  entire  Method  3  consists  of  Parts  I  and  II  of  Method  1  (which 
obtain  all  possible  improvements  by  changing  only  one  variable  at  a  time) 
plus  the  above  procedure  as  a  replacement  for  Parts 
III  to  VII  (which  investigate  certain  ways  of  changing  two  variables 
simultaneously).  Thus,  after  completing  Part  II,  set  =  x  and 

go  into  the  above  search  procedure.  If  a  better  feasible  solution  is 
found  (i.e.,  q  <  0  in  Step  5  for  some  iteration  of  some  cycle),  then 
immediately  restart  Phase  3  at  the  beginning  of  Part  II  with  this  new 
solution.  However,  if  the  search  procedure  goes  through  all  n  cycles 
without  finding  a -better  feasible  solution  (perhaps  on  a  later  time  through), 
then  Phase  3  terminates  with  the  current  x^  as  the  desired  approximate 
solution. 

Method  k  differs  from  Method  3  only  in  that  all  of  the  original 
Phase  3  (Method  l)  is  completed  before  entering  the  above  search  procedure. 
(This  holds  on  both  the  first  time  and  subsequent  times  through  the  overall 
process.)  Thus,  Method  1  is  guaranteed  to  do  at  least  as  well  as  Method  1 
in  obtaining  a  good  final  solution,  but  it  may  require  substantially  more 
time  than  Method  3  without  a  significant  increase  in  effectiveness. 

A  serious  drawback  of  the  search  procedure  used  by  both  Methods  3 

2 

and  4  is  that  it  requires  more  than  some  multiple  of  mn  elementary 
operations,  so  that  the  time  required  grows  rapidly  with  the  sice  of  the 
problem  (more  so  than  the  rest  of  the  procedure).  Therefore,  Method  5 
modifies  Method  4  by  streamlining  the  search  procedure.  In  particular, 
rather  than  n  cycles,  there  is  only  one.  Furthermore,  no  change  is 
made  in  before  starting  the  modified  Steps  4  to  9  of  Phase  2. 


10 


In  order  to  avoid  thereby  terminating  immediately  because  no  "improvement" 
is  possible,  Steps  7  and  8  are  replaced  by  the  following.  Calculate 
p  ,  p.,,  . .  .  ,  p^  in  the  usual  way.  (Again,  criterion  A  was  used  for  these 

calculations  in  the  testing  program.)  Set  k  equal  to  a  value  such  that 

p  =  max  p.,  regardless  of  whether  this  maximum  is  positive  or  not,  and 
X  1 

then  go  to  Step  9.  Thus,  at  each  iteration,  the  variable  is  changed  which 
has  the  best  effect  on  the  "infeasibility"  q,  even  though  it  may  actually 
increase  q.  As  a  result,  some  drifting  away  from  the  reduced  feasible 
set  may  occur  before  the  search  can  (hopefully)  home  in  on  an  improved 
feasible  solution.  To  avoid  drifting  indefinitely,  an  upper  limit  is 
imposed  on  the  number  of  iterations,  which  was  taken  to  be  100  for  purposes 
of  computational  testing.  Another  danger  of  eliminating  the  requirement 
that  a  must  be  strictly  reduced  at  each  iteration  is  that  it  is  then 
possible  to  begin  cycling,  whereby  the  same  sequence  of  solutions  is 
repeated  ad  infinitum.  One  inexpensive  way  to  lesson  the  risk  of  this 

occurring  is  to  impose  the  restriction  that  a  variable  change  cannot  be 
reversed  within  a  certain  number  of  iterations  (taken  to  be  five  for 
computational  testing) . 

Because  of  its  ability  to  drift,  it  is  conceivable  that  Method  5 
actually  would  be  more  effective  than  Methods  3  ana  4  in  reaching  a  better 
feasible  solution  that  requires  changes  in  many  variables. 

c •  Description  of  Test  Problems 

As  described  previously  by  the  author  [9],  some  38  test  problems 
-ere  used  to  evaluate  his  original  heuristic  procedures.  Most  of  these 
were  Type  I  and  Type  II  problems  as  described  in  Table  I,  vrhere  the 
respective  parameters  are  integers  randomly  generated  (using  the  mixed 
congruential  method)  from  a  (discretized)  uniform  distribution  over  the 
indicated  interval.  The  Type  I  problems  are  intended  to  be  difficult 
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problems  that  should  be  particularly  challenging  for  the  heuristic  procedures 
since  there  is  no  exploitable  special  structure,  the  coefficient  matrix  is 
completely  dense,  feasible  solutions  are  relatively  difficult  to  identify, 
and  the  variables  have  a  wide  range  of  values  to  be  considered.  Thus,  these 
problems  should  be  exceptionally  effective  in  revealing  any  significant 
differences  in  the  effectiveness  of  alternative  procedures.  The  Type  II 
problems  are  intended  to  be  representative  of  the  multidimensional  knapsack 
problems  (all  coefficients  nonnegative)  frequently  found  in  practice.  See 
[8]  for  a  listing  of  these  original  test  problems  of  both  types  (except  those 
larger  than  30  x  30) . 

TABLE  I 

DESCRIPTION  OF  THE  RANDOMLY  GENERATED  TEST  PROBLEMS 


Parameter 

Problem  Type 

I 

la 

Ic 

II 

c  . 

J 

(-20,791 

Type  I 

Type  I 

(0,991 

a.  . 

in 

[  -40,593 

except 

except 

[0,991 

b. 

1 

(500,9991 

■  °>  -  E 

P(c.  =  0)  =  \ 

[1000,19991 

_ 

Eighteen  of  these  original  Type  I  and  Type  II  problems  were  used 
again  to  test  the.  new  methods  presented  in  the  preceding  sections.  These 

consisted  of  the  16  problems  with  m  =  15,  n  =  15  for  which  the  standard-of- 
comparison  procedure  (1-2A-1)  definitely  did  not  obtain  an  optimal  solution 
(namely,  problems  1-2  through  1-8  and  11-1,3,4,7,9,11,13,14,15)  plus  two 
larger  problems  for  which  the  optimal  solution  is  unknown  (11-16,  which  is 
30  X  30,  and  the  30  X  60  "Large"  Type  I  problem,  labeled  1-9  here) . 

In  addition,  4o  new  test  problems  with  m  =  10,  n  =  20  were  generated 


at  the  outset.  These  include  20  ordinary  Type  I  problems  (labeled  I-I01 
through  1-120),  plus  10  Type  la  problems  (la-1  to  la-10)  and  10  Type  Ic 
problems  (lc-1  to  Ic-10)  as  described  in  Table  I.  Specifically,  for  la 


problems,  each  was  assigned  a  value  of  2ero  with  probability 

if  this  event  did  not  occur,  then  it  was  assigned  a  randomly  generated 

integer  (possibly  zero)  from  the  interval  [-^0,59]  in  the  usual  way. 

For  Ic  problems,  the  c.  parameters  were  generated  in  an  analogous  way. 

J 

Since  "low  density"  problems  (particularly  with  respect  to  the  a. .)  are 

t  J 

commonly  observed  in  practice,  these  la  and  Ic  problems  were  intended 
to  check  on  the  effect  cf  this  factor.  No  additional  Type  II  problems  were 
generated  since  these  had  proven  to  be  far  less  challenging  (i.e.,  the 
solutions  obtained  tended  to  be  far  closer  to  being  optimal)  than  the  Type  I 
problems  for  the  original  procedures.  (None  of  the  original  Type  III  problems 
were  used  for  the  same  reason.) 

Finally,  to  avoid  complete  reliance  on  randomly  generated  problems, 
use  was  made  of  the  nine  IBM  problems  presented  by  Haldi  [6]  and  reproduced 
by  Trauth  and  Woolsey  [183*  (None  of  Haldi's  other  test  problems  were  used  since, 

as  reported  in  [9],  two  of  the  most  difficult  ones  had  presented  little  challenge 
to  the  heuristic  procedures  in  previous  testing.) 

It  should  be  noted  that  all  67  of  these  test  problems  are  relatively  small 
ones  compared  to  the  sizes  that  would  be  computationally  feasible  for  the 
heuristic  procedures.  The  two  reasons  for  doing  this  were  to  preserve  a  fairly 
limited  computer  budget  and  to  permit  identifying  an  optimal  solution  with  an 
exact  algorithm  for  most  cases.  The  emphasis  in  this  testing  program  was  on 
evaluating  the  effectiveness  of  the  procedures,  as  measured  by  the  normalized 
deviation  from  optimality  (defined  below).  Some  information  also  was  obtained 
on  their  efficiency,  but  no  attempt  was  made  to  test  their  limits  of  computational 
feasibility. 

All  testing  was  done  on  an  IBM-360/ 67  computer,  using  FORTRAN  codes. 
Documentation  of  the  code  for  the  original  procedures  is  available  in  a  separate 
report  [11]. 


6.  Evaluation  of  Some  Original  Procedures  and  the  Phase  2  Criteria 


6.1.  Test  Results 


The  experimentation  program  began  by  dealing  with  the  old  unresolved 
question  of  which  basic  type  of  procedure  (from  among  1-2-1,  2-2-1,  l-J-l, 
and  2-3-1),  in  conjunction  with  which  Phase  2  criterion  (A,  B,  C,  D,  plus 
the  new  E  and  S),  is  best.  Thus,  24  distinct  procedures  were  to  be  con¬ 
sidered.  Since  these  procedures  (except  with  the  new  Phase  2  criteria) 
already  had  been  applied  (see  Table  II  ir.  [9])  to  the  18  old  test  problems 

(except  for  1-9),  all  24  now  were  individually  run  on  just  the  49  new  test 
problems  plus  1-9. 


On  21  of  these  50  problems,  the  same  final  solution  was  obtained  by 

all  24  procedures.  The  results  for  the  other  29  problems  are  shown  in 

Table  II.  For  each  problem,  this  table  compares  the  final  solution  obtained 

by  each  procedure  (identified  partially  by  the  footnotes  to  the  table)  with  the 

best  solution  obtained  by  any  of  the  24  procedures,  expressed  in  terms  of  the 

normalized  differences  in  their  objective  function  values.  Specifically,  if 

xq  denotes  the  objective  function  value  of  this  best  value,  then  the  normalized 

deviation  of  a  given  solution  with  objective  function  value  x  from  this 

o 

best  solution  is 


B  I  n  2 

normalized  deviation  =  (xq  -  xo)/|  £  c.. 


1 

2 


so  that  this  normalized  deviation  is  just  the  Euclidean  distance  between 
the  hyperplanes, 


n 

£ 

j=i 


C  .X , 

0  j 


x®  and 
o 


n 

E 

0=1 


c  .X .  = 
0  0 


x 

0 


(When  the  optimal  solution  is  known,  substituting  its  objective  function 
value  for  xq  in  the  above  expression  yields  the  normalized  deviation  from 
optimality,  which  is  the  quantity  used  in  several  of  the  subsequent  tables.) 


For  many  of  the  problems,  the  four  basic  types  of  procedures  obtained 
the  same  final  solution  for  a  given  Phase  2  criterion,  in  which  case  the 
resulting  normalized  deviation  is  listed  singly  in  the  table.  VThen  these 
procedures  obtained  different  solutions,  the  resulting  normalized  deviations 
are  listed  separately  with  footnotes  identifying  the  procedures  involved. 

The  average  normalized  deviation  for  each  procedure  is  given  at  the  bottom 
of  the  table,  where  this  average  first  excludes  just  IBM- 7  (since  criteria 
A,  B,  and  D  failed  to  find  any  feasible  solution  for  this  problem)  and 
next  excludes  both  IBM-7  and  IBM-8  (since  the  IBM-8  results  were  dominating 
the  first  average). 

Since  the  results  shown  in  Table  II  were  inconclusive  in  identifying 
the  best  procedure,  a  supplementary  experimental  program  involving  2ko 
additional  test  problems  was  undertaken  subsequently,  as  described  in  the 
appendix. 

In  addition  to  the  information  in  Table  II,  various  other  performance 
data  also  were  gathered  for  the  50  test  problems.  Since  these  data  tend 
to  vary  with  problem  size,  they  are  summarized  on  an  average  basis  in 
Table  III  for  just  the  4o  problems  where  m  =  10,  n  =  20.  The  last  three 
columns  give  a  grand  average  over  all  six  Phase  2  criteria  for  the  indicated 
basic  type  of  procedure,  expressed  as  an  increment  over  the  corresponding 
grand  average  for  the  1-2-1  procedure.  Letting 

x  =  (l-a)x^  +  ax^  , 


the  first  set  of  rows  refers  to  the  value  of  a  at  which  a  feasible  solu¬ 
tion  is  found  during  the  subsequent  Phase  2  search.  The  next  set  of  rows 

,  ( t ) . 

refers  to  the  number  of  points  (excluding  x  )  on  this  line  segment 
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NORMALIZED  DEVIATION  FROM  BEST  SOLUTION  FOR  PROBLEMS  WITH  DIFFERENT  SOLUTIONS 


Used  in  Phase 


10  !20  1-118 


TABLE  II 
(Continued) 


Criterion  Used  in  Phase 


AVERAGE  OF  PERFORMANCE  DATA  FROM  10  X  20  PROBLEMS  FOR  EACH  PHASE  2  CRITERION 
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between  x  and  xv  that  are  used  to  launch  a  Phase  2  search  until 

this  feasible  solution  is  found.  The  number  of  "trial  solutions"  refers 

to  solutions  that  are  generated  during  these  Phase  2  searches  (step  9) 

as  possible  feasible  solutions,  excluding  rounded  solutions  from  the 

line  segment.  Normalized  deviation  is  defined  just  as  above,  with  xq 

now  replaced  by  the  objective  function  value  for  x^ .  The  two  modes 

of  search  used  in  Method  1  of  Phase  3  involve  charging  just  one  variable 

at  a  time  (Part  II)  and  then  changing  pairs  of  variables  simultaneously. 

Thus,  "Normalized  improvement,  first  time  in  Part  II  of  Phase  3"  shows 

(1) 

the  decrease  in  normalized  deviation  from  x  that' would  result  if 
only  the  first  node  of  search  were  used  to  improve  upon  the  original 
feasible  solution,  whereas  the  next  set  of  rows  shows  the  additional 
improvement  from  using  the  full  Phase  3-  "Total  tine  for  procedure" 

consists  of  the  total  CPU  time  (in  seconds)  for  the  entire  heuristic  procedure 
except  the  time  required  by  the  simplex  method  to  obtain  x^  (which  may  be 
several  times  as  large  as  for  the  heuristic  procedure).  No  special  provisions 
were  made  for  controlling  timing  variability  on  the  computer,  so  the  times 
obtained  for  individual  problems  may  have  a  substantial  variance  due  solely  to 
this  factor,  although  the  corresponding  variance  for  the  average  times  over  40 
problems  recorded  in  Table  III  would  be  very  much  smaller. 

6.2.  Comparison  of  Basic  Types  of  Procedures 

Tables  II,  III,  IX,  and  X  (see  the  appendix)  provide  the  primary 
basis  for  comparing  the  four  basic  types  of  procedures,  1-2-1,  1-3-1, 

2-2-1,  and  2-5-1.  These  rather  extensive  results  fail  to  reveal  any 
significant  differences  in  the  effectiveness  of  1-2-1,  1-5-1,  and  2-3-1 


as  measured  by  the  normalized  deviation  from  the  best  solution.  However, 
they  do  show  2-2-1  rather  consistently  lagging  behind  the  other  three  on  the 
average  (see,  in  particular,  the  averages  given  at  the  bottom  of  Tables  II,  IX 
and  X).  Furthermore,  the  time  comparison  in  Table  III  indicates  that  2-2-1  has 
far  too  slight  an  advantage  in  efficiency  to  be  able  to  compensate  in  this  way 
for  these  substantial  differences  in  effectiveness. 

Some  of  the  details  in  Table  III  may  provide  an  explanation  for  the 

relatively  weak  performance  of  2-2-1.  Kote  that  the  average  "Xum.ber  of 

points  tried  until  a  feasible  solution  is  found"  for  2-2-1  is  only  about 

•i  or  less  that  for  the  other  procedures.  On  the  other  hand,  specific 

comparison  with  2-3-1  on  this  statistic  ana  on  the  average  "Value  of  cx 

at  which  a  feasible  solution  is  found"  suggests  that  2-2-1  has  moved 

somewhat  further  than  the  other  procedures  along  the  line  segment  from 

x^  to  x^  before  finding  a  feasible  solution.  (The  a  values  cannot 

(2) 

be  directly  compared  between  1-x-x  and  2-x-x  procedures  since  the  x 

( 1 ) 

from  Method  2  of  Phase  1  tends  to  be  considerably  further  away  from  x 
than  for  Method  1.)  Thus,  2-2-1  evidently  skips  over  many  rounded  solu¬ 
tions  near  this  line  segment  that  would  be  considered  by  the  other  pro¬ 
cedures.  The  result  was  that  the  feasible  solution  obtained  in  Phase  2 
was  considerably  inferior  to  those  for  the  other  procedures  on  the  average 
(see  the  data  in  the  fourth  set  of  row's),  and  Phase  3  was  not  fully  able 
to  recoup  this  deficit. 

This  suggests  that  the  quality  of  the  final  solution  is  affected 
significantly  by  the  proximity  to  when  initiating  the  Phase  2 

search  that  leads  eventually  to  this  solution  at  the  culmination  of  Phase  3* 
Therefore,  when  moving  from  x^  toward  x^^  in  Phase  2,  it  is  important 
to  try  not  to  skip  over  points  that  may  successfully  lead  to  a  leasible 
solution.  In  other  words,  it  appears  that  a  x-3-x  procedure  should  tend 
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to  be  at  least  slightly  more  effective  on  the  average  than  the  corresponding 
x-2-x  procedure.  (it  also  tends  to  require  a  little  more  time  on  the 
average. ) 

Although  this  difference  in  effectiveness  did  not  become  apparent 

in  the  comparison  between  1-2-1  and  1-3-1,  a  plausible  explanation  can 

again  be  found  in  Table  III.  Specifically,  the  second  set  of  rows  suggest 

that  the  jumps  along  the  line  segment  being  made  in  Phase  2  of  1-2-1  were 

small  enough  (except  on  the  la  problems)  to  avoid  skipping  over  very  many 

points  that  were  being  tried  by  1-3-1.  However,  there  is  no  reason  to 

expect  comparable  jump  sizes  in  general,  particularly  since  the  number  of 

distinct  rounded  solutions  generated  by  the  line  segment  between  and 

(2) 

x  tends  to  increase  with  the  number  of  variables  in  the  problem. 

Because  x-5-x  procedures  do  have  considerable  variability  in  the 
number  of  points  tried  and,  thus,  in  the  total  time  for  Phase  2,  it  might 
prove  worthwhile  on  very  large  problems  to  use  a  compromise  between  an 

x-3-x  and  an  x-2-x  procedure.  In  other  words,  one  could  begin  by  selecting 
points  on  the  line  segment  between  x^1^  and  according  to  the 

x-3-x  procedure,  but  after  a  certain  number  of  futile  tries,  one  could 
then  switch  over  to  using  x-2-x.  (More  complicated  ways  of  merging  the 
two  approaches  also  could  be  devised.) 

Comparing  1-x-x  and  2-x-x  procedures  is  inherently  very  difficult 
since  differences  can  arise  on  a  given  problem  only  when  a  >  0,  which 
frequently  does  not  occur,  and  the  differences  are  not  likely  to  be  major 
ones  (on  the  average)  unless  a  is  quite  large,  which  seldom  happens. 

The  numerous  test  problems  run  here  certainly  provide  little  basis  for 
choosing  between  1-3-1  and  2-5-1.  However,  there  are  some  clues  in  Table 
III  that  may  be  significant.  In  particular,  the  second,  third,  and  fourth 


sets  of  rows  indicate  that,  for  every  type  of  problem,  1-3-1  needed  to  try 
less  points  and  less  trial  solutions  in  order  to  obtain  a  better  feasible 
solution  (on  the  average)  than  2-3-1.  This  suggests  that  Method  1  of 
Phase  1  may  give  a  more  centralized  route  into  the  interior  of  the  feasible 
region  that  more  readily  leads  to  a  good  feasible  solution  than  does  Method 
2.  Additional  analysis  by  Faaland  and  Hillier  [2]  also  suggests  that  this 
is  the  case.  Therefore,  in  the  absence  of  contravening  evidence,  it  is 
recommended  that  1-3-x  be  adopted  as  the  preferred  choice  from  among  the 
four  basic  types  of  procedures  tested. 

It  also  should  be  noted  that  Faaland  and  Hillier  [3]  have  analyzed  the 
present  Phase  1  methods  from  a  statistical  viewpoint.  This  analysis  led  them 
to  propose  some  promising  new  modifications  and  extensions  of  these  methods, 
including  the  use  of  a  piecewise  linear  path  between  x^  and  The 

results  of  some  limited  comparative  testing  also  are  presented. 

6.3.  Comparison  of  Phase  2  Criteria 

The  same  four  tables  (II,  III,  IX,  and  X),  plus  the  additional  data 
and  statistical  analysis  summarized  in  Table  XI  of  the  appendix,  provide 
the  primary  basis  for  comparing  the  six  Phase  2  criteria  being  considered 
here. 

Criteria  A  and  B  are  the  only  two  that  do  not  consider  the  objective 
function,  since  they  are  based  entirely  on  (different)  measures  of  infeasi¬ 
bility.  Comparing  these  two  criteria  first,  the  tables  show  Criterion  B 
often  lagging  substantially  behind  Criterion  A  (particularly  with  the 
preferred  1-3-1  procedure).  Since  these  criteria  differ  only  in  their 
measure  of  infeasibility,  it  appears  that  A's  measure  probably  is  a  more 
effective  one  than  B’s. 
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Criteria  C  and  D  use  the  same  measure  of  improvement  x>,  which  does 
take  the  objective  function  into  account  in  a  certain  way.  Otherwise, 
Criterion  C  is  identical  to  A,  and  Criterion  D  is  identical  to  B.  There¬ 
fore,  to  be  consistent  with  the  above  conclusion  on  A  and  B,  Criterion  C 
probably  should  be  preferred  to  D.  (The  data  do  not  reveal  this  difference 
clearly,  since  Criterion  D  performs  very  strongly  in  Tables  II  and  III,  whereas 
Criterion  C  does  a  little  better  in  Tables  IX  and  X.) 

The  appendix  describes  how  200  additional  test  problems  were  used 
to  try  to  distinguish  between  the  four  remaining  criteria  -  A,  C,  E,  and 
S.  However,  even  this  amount  of  testing  was  unable  to  detect  differences 
at  a  reasonable  level  of  statistical  rigr.ificance.  The  main  conclusion 
seems  to  be  that,  even  though  large  differences  can  occur  on  individual 
problems,  the  choice  of  the  criterion  does  not  have  a  strong  effect  on  the 
average  performance  of  the  heuristic  procedure  in  the  long  run.  All  of 

the  available  evidence  does  suggest  that  the  new  Criterion  S  may  be  some¬ 
what  inferior  to  the  others,  but  that  the  new  Criterion  E  may  be  at  least 
as  good  as  any  of  the  others. 

Table  III  indicates  that  Criterion  S  does  achieve  its  objective  of 
substantially  reducing  the  time  for  finding  a  feasible  solution.  However, 
since  Phase  3  tends  to  require  somewhat  more  time  than  Phase  2,  the  pro¬ 
portional  reduction  in  the  total  time  for  the  heuristic  procedure  is 
relatively  modest.  This  should  tend  to  be  the  case  for  much  larger  problems 
as  well,  except  when  the  "Number  of  points  tried  until  a  feasible  solution 
is  found"  becomes  large,  which  sometimes  would  occur  with  a  x-3-x  type 
of  procedure. 

Table  III  also  reveals  two  other  interesting  contrasts  between 
Criterion  S  and  the  others.  First,  with  its  short  and  undiscriminating 
moves  toward  feasibility,  Criterion  S  tends  to  more  quickly  find  a  feasible 


solution  in  the  sense  of  doing  so  with  a  smaller  value  of  ct,  and  so  with 
fever  points  tried  along  the'  line  segment  from  x^  to  .  At  the 

same  time,  the  resulting  feasible  solution  tended  to  be  inferior  to  those 
obtained  by  the  other  criteria.  Therefore,  if  the  prime  objective  is  to 
obtain  the  best  possible  feasible  solution,  then  it  seems  better  to  be 
more  patient  and  discriminating,  as  vrith  the  other  criteria.  (The  last  two 
sets  of  rows  in  Table  III  indicate  that  the  other  criteria  average  requiring 
almost  twice  as  much  time  in  Phase  2  as  Criterion  S,  but  that  the  proportional 
difference  in  the  resulting  total  time  for  the  overall  procedure  is  considerably 
less.)  Second,  the  other  criteria  have  a  considerably  smaller  ratio  of  "number  of 
trial  solutions"  to  "number  of  points  tried"  than  Criterion  S,  so  they  tend  to 
get  blocked  from  making  further  moves  toward  feasibility  fairly  quickly 
and  readily.  Therefore,  on  problems  where  it  is  relatively  difficult  to 
find  any  feasible  solution,  it  appears  that  Criterion  S  may  be  more  effective 
in  actually  reaching  such  a  solution. 

7 .  Evaluation  of  Phase  3  Methods 
7.1.  Test  Results 

The  next  step  in  the  experimental  program  was  to  test  the  new  methods 
for  Fhase  5  described  in  Section  k.  Using  a  fixed  Phase  1  and  2  (Methods 
1  and  2A,  respectively),  these  three  methods  plus  the  old  Method  1  were 
applied  to  the  18  old  problems  of  Types  I  and  II  identified  above,  the 
nine  IBM  problems,  and  (for  economy  reasons)  just  the  first  four  new  problems 
of  each  of  Types  I,  la,  and  Ic.  The  resulting  normalized  deviation  of  the  final 
solution  from  the  optimal  solution  is  shown  in  Table  IV,  along  with  the  total 

time  used  in  Phase  3*  When  the  optimal  solution  was  not  known,  a  lower 
bound  on  the  normalized  deviation  from  optimality  is  shov.'r,  instead,  pre¬ 
ceded  by  a  >  sign.  (Since  the  three  phases  are  not  independent,  note  that 
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Type  &  No. 


1-2 
1-5 
1-1 
1-5 
1-6 
1-7 
1-8 
II-l 
II-5 
II -1 
II-7 
II-9 
II- 11 
11-15 
II- ll 
11-15 
II-16 
I -101 
1-102 
1-105 
i-iol 

la-1 

la-2 

la -5 
la-1 
Ic-1 
Ic-2 

Ic-5 


1-2A-1  1-2A-5  1-2A-1 


I  0.041 


>  0.208 


hj 


Total  Time 
for  Phase  5 


1-2A-5  |1-2A-1  1-2A-5  1-2A-1 


0.181  0.056 

O.169  0.076 

0.192  0.285 

0.229  0.175 

0.018  0.109 

0.011  0.0H 

0.170  0 

0.056  0 
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0.015  0.015 

0.018  0.018 

0.056  0.056 

0.012  0 
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0.101 
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• 

O 
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0 
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0.078 
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0 
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0 

0 

0.052 

0.015 
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0 
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0 
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>  0.017 
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0.17  0..25 

0.52  0.15 
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TABLE  IV 
(Continued) 


m 

n 

Problem 
Type  &  No. 

Normalized  Deviation  ! 

from  Optimality  j 

- - - L 

t  Total  Time 

|  for  Phase  3 

1-2A-1 

1-2A-3 

1-2A-4 

1-2A-5  | 
• 

1-2A-1 

1-2A-3 

1-2A-4 

1-2A-5 

10 

20 

Ic-4 

>  0.155 

>  0.036 

>  0.036 

>  0 

0.14 

0.51 

1.25 

0.74 

7 

7 

IBM-1 

0.378 

0 

0 

0 

0.04 

0.09 

0.11 

0.44 

7 

7 

IBM-2 

0 

0 

0 

0 

0.02 

0.03 

0.05 

0.34 

3 

4 

IBM-3 

0 

0 

0 

0 

0.03 

0 

0.05 

0.02 

15 

15 

IBM-4 

0.258 

0.258 

0.258 

0.258 

0.14 

0.39 

0.81 

1.66 

15 

15 

IBM- 5 

0.258 

0.258 

0.258 

0.258 

0.08 

0. 47 

0.88 

1.55 

31 

31 

IBM-6 

0.360 

0.180 

0.180 

0.180 

0.38 

O.bO 

0.8l 

5 .6i 

12 

50 

IBM-7* 

o' 

0.075 

0 

0 

0.56 

1.85 

'  2.27 

3-36 

12 

37 

IBM-8 

13 

13 

0 

0 

0.05 

1.11 

0.79 

0.13 

50 

15 

IBM-9 

0 

0 

0 

0 

0.15 

0.84 

0.91 

4.03 

30 

6o 

1-9 

1 

>  0.465 

>  0.465 

>  0.459 

2.01 

1.86 

20.91 

23.33 

Average 

>  0.610 

>  0.41*9 

>  0.104 

>  0. 078 

0.37 

0.62 

I.69 

3-00 

Average  without 
IBM-8 

>  0.283 

>  0.119 

>  0.107 

>  0.080 

0.38 

0.6l 

1.71 

3-03 

Criterion  C  was  used  on  this  problem  since  Criterion  A  did  not  give  a  feasible 
solution  in  Phase  2. 
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the  relative  performance  of  the  Phase  3  methods  could  change  somewhat  if  a 
different  combination  of  methods  for  Phases  1  and  2  were  used;  budget  limitations 
prevented  checking  this  further.) 

72.  Comparison  of  Phase  3  Methods 

The  last  comment  of  Sec.  6.3  actually  has  a  bearing  on  comparing  Phase  3 
methods  also.  Method  5  follows  a  philosophy  similar  to  Criterion  S  for  Phase  2 
in  that  it  is  relatively  undiscriminating  in  the  individual  moves  it  allows  in  order 
to  expedite  making  combinations  of  moves  that  may  indeed  lead  to  good 
feasible  solutions.  By  contrast,  Methods  1,  3,  and  k  are  analogous  to 
the  other  criteria  in  that  they  set  high  requirements  for  each  individual 
move,  and  so  tend  to  get  blocked  from  making  further  moves  toward  improved 
feasible  solutions  fairly  quickly  and  readily.  The  results  of  Table  IV 
suggest  that  the  less  discriminating  approach  of  Method  5  tends  to  be 
more  effective  in.  actually  reaching  improved  feasible  solutions.  However, 
the  much  longer  (albeit  fewer)  searches  involved  consume  considerable 
time,  and  Table  IV  indicates  that  the  increase  in  total  time  over  the  other 
methods  tends  to  be  substantial.  In  particular,  Method  5  has  an  Average  Total 
Time  for  Phase  3  in  Table  IV  that  is  approximately  2,  5  and  8  times  as  large  as 
that  for  Methods  4,  3  and  1,  respectively. 

Method  3  emerges  as  a  "best  buy"  approach  in  terms  of  the  trade-off 
between  execution  time  and  the  quality  of  the  solution  attained. 

The  growth  rates  of  total  time  with  problem  size  in  Table  IV  appear 
to  be  roughly  comparable  for  the  different  methods.  However,  there  may 
be  small  differences  in  the  growth  rates,  so  it  is  uncertain  whether  the  time 
comparisons  between  methods  observed  here  would  still  hold  for  very  large  problems. 

Finally,  note  that  the  fifth  and  sixth  set  of  rcrws  in  Table  III  show 
that  going  on  to  the  second  mode  of  search  in  Method  1  tends  to  be  very 
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worthwhile,  more  than  doubling  the  improvement  from  just  the  first  mode 
of  search  alone  (except  for  Ic  problems)  on  the  average.  However,  the 
fact  that  Method  5  performed  nearly  as  well  as  Method  4  in  Table  17  demon¬ 
strates  that  little  is  lost  by  foregoing  the  second  mode  of  search  if 

the  new  mode  introduced  by  these  methods  also  is  being  used.  In  fact, 
the  second  mode  of  search  achieved  a  larger  improvement  than  the  new 
mode  used  in  Method  3  (or  4)  on  only  3  of  the  39  problems,  and  was  out¬ 
performed  17  times.  Furthermore,  Method  4  (which  uses  both  the  second 
mode  and  the  new  mode)  achieved  a  further  improvement  over  the  second 
mode  on  15  of  the  35  problems  where  optimality  had  not  yet  been  reached. 
Nevertheless,  comparing  Methods  4  and  5  in  Table  IV  shows  that  this  new 
mode  itself  was  outperformed  by  the  new  mode  used  in  Method  5  on  14  of 
the  19  problems  where  differences  occurred.  The  Method  5  mode  of  search 
actually  achieved  a  further  improvement  over  the  second  mode  25  out  of 
the  35  possible  times. 

The  unescapable  conclusion  is  that  changing  only  one  or  two  variables 
is  frequently  inadequate  for  reaching  a  better  feasible  solution.  It 
may  be  necessary  to  change  many  variables.  Since  it  wouldn't  be  compu¬ 
tationally  feasible  to  investigate  such  simultaneous  changes  directly, 
one  needs  a  mode  of  search  that  accomplishes  this  indirectly  by  making 
a  long  sequence  of  small  changes.  Some  of  these  small  changes  may  need 
to  worsen  the  situation,  when  considered  individually,  in  order  to  permit 
combinations  of  changes  which  provide  a  total  overall  improvement,  a  la 
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8.  Evaluation  of  Multiple  Solution  Procedures 
8.1.  Test  Results 

The  investigation  next  considered  the  question  of  the  best  way  to 
generate  multiple  solutions  in  order  to  try  to  improve  upon  the  -first 
final  solution.  The  first  such  method  tested  was  the  one  proposed  in  [9  ] 
of  repeating  the  Phase  2  search  over  the  entire  line  segment  between 
x^  and  x^  in  order  to  generate  a  series  of  feasible  solutions 
as  starting  points  for  Phase  3.  The  results  of  doing  this  with  Method  3 
of  Phase  2  (which  provides  the  most  exhaustive  possible  search  over  this 
line  segment)  are  shown  in  Table  V  for  Procedures  1-R3-1  and  2-R3-1  under 
all  six  Phase  2  criteria. 

Procedure  x-R3x-l  is  guaranteed  to  do  at  least  as  well  as  the  corres¬ 
ponding  x-2x-l  (of  x-3x-l)  procedure  (since  it  obtains  the.  same  solution 
as  well  as  others),  and  it  sometimes  will  provide  a  significant  improvement. 
In  fact,  for  the  problems  in  Table  V  where  the  x-2x-l  procedure  did  not 
obtain  an  optimal  solution,  x-R3x-l  gave  an  improvement  on  essentially 
half  of  them,  and  the  average  fractional  improvement  (when  it  did  occur) 
on  the  original  normalized  deviation  from  optimality  was  approximately  i. 

Table  VI  provides  a  more  detailed  analysis  of  this  approach  for  the 

1-R3A-1  procedure.  The  number  of  distinct  solutions  in  Phase  1  refers 

to  the  number  of  rounded  solutions  that  were  obtained  from  the  line  segment 

(l)  (2) 

between  x  and  x  .  Each  of  these  new  rounded  solutions  initiates 
a  Phase  2  search  for  a  feasible  solution,  and  the  ”Ph.2”  column  gives  the 
number  of  times  this  search  actually  obtained  a  new  feasible  solution. 

Since  each  such  solution  leads  to  a  final  solution  from  Phase  3,  the 
"Ph.3"  column  shows  how  many  distinct  final  solutions  were  thereby  obtained. 


The  "First  Solution"  column  gives  the  range  of  a  that  would  successfully 
lead  to  the  first  final  solution  obtained.  The  "Best  Solution"  column 
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TABLE  VI 

SUMMARY  OF  PERFORMANCE  FOR  PROCEDURE  1-R3A-1 


Problem 
Type  &  No. 


No.  of 
Distinct 
Solutions 


Ph.l  Ph.2  Ph.3 


10  120 


1-5 
1-6 
1-7 
1-8 
.  II-l 
II-3 
II-4 
II-7 
II-9 
11-11 
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0.236-0.586 

0. 

131-0.256 

0.352-0.405 

Best  Solution 


a 


0.782-0.801 


Deer,  in 
Norm.  Dev. 


0  -0.572 

0.646-1 

0.082-0.389 

0.668-0.828 

0.129-0.870 

0.752-1 
0  -0.153 

0.942-1 
0  -1 

0.392-1 
0  -1 

0  -0.660 

0  -1 

0  -1 

0.209-1 


Ph.2  Sol—  I  from  Opt. 


#4 


#2-3 

#6-7 


0.585-0.594 

#3 

0.110 

0  -0.135 

#1  1 

0 

0  -0.081 

#1 

0.350-0.402 

#6 

0 

0.288-0.327 

#2 

0.022 

0.236-0.586 

#1-2 

0 

0.257-0.310 

#2 

0.280 

0.799-0.801 
0.865-1  | 

#5 

#8 

0.208 
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Type  &  No, 
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50  37  24  0 


First 
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Best  Solution 
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#1 

0 
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#2 

0 
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0  -1 
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0 
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0 
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#2-4 
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0 

-0. 4i8 

0  -0.4l8 

#1-2 
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0.221-0.284 

#3 

#5 

0.046-0.512 

0.209-0.698* 

#2. 1-2. 9* 

Deer,  in 
Norm.  Dev. 


1-9 


Average 


When  more  than  one  interval  for  a  yielded  the  best  solution,  the  widest  interval 
was  used  for  purposes  of  calculating  the  average. 

Linear  extrapolation  was  used  to  estimate  "No.  of  distinct  solutions"  si:  'e  the 
run  was  terminated  before  a  =  1. 


shows  the  range  of  a  that  would  lead  to  the  best  final  solution  obtained, 
and  then  identifies  which  of  the  distinct  feasible  solutions  obtained 
in  Phase  2  (in  chronological  order  of  increasing  a)  led  to  this  best 
solution.  The  final  column  shows  the  resulting  decrease  in  the  normalized 
deviation  from  optimality  compared  to  the  1-2A-1  procedure. 

After  making  two  observations,  Table  VI  also  indicates  how  the  computa¬ 
tional  effort  of  1-R3A-1  on  these  problems  compares  with  that  for  1-3A-1 

(or  1-2A-1).  First,  the  time  required  for  Phase  1  (other  than  obtaining  x^ 

by  the  simplex  method)  is  negligible  compared  to  Phases  2  and  3.  Second,  the 
time  required  to  go  through  Phases  2  and  3  again  to  generate  a  new  (not 

necessarily  distinct)  final  solution  tends  to  be  about  the  same  as  for  obtaining 

the  initial  final  solution  with  1-3A-1  (or  1-2A-1) .  Therefore,  the  "No.  of 
Distinct  Solutions  in  Ph.  2"  column  indicates  approximately  the  multiple  of 
the  time  for  1-3A-1  (excluding  the  simplex  method)  that  was  required  for  1-R3A-1. 

Table  VII  summarizes  the  average  performance  of  the  x-R3x-l  procedures 
on  these  same  39  problems  for  all  12  combinations  of  Phase  1  methods  and 
Phase  2  criteria.  In  addition  to  showing  the  same  type  of  data  as  in 
Table  VI  (where  "Deer,  in  Norm.  Dev.  from  Opt."  now  is  in  comparison 
with  the  corresponding  x-2x-l  procedure),  the  last  two  sets  of  rows  con¬ 
trasts  the  results  being  obtained  at  those  times  when  the  .procedure  currently 
is  initiating  the  Phase  2  search  from  the  two  extreme  points  (a  =  0,  1) 
and  from  the  equivalent  of  the  midpoint  (a  =  i)  of  the  line  segment 
between  x^  and  The  average  times  given  for  these  three  cases 

include  just  the  time  required  for  Phase  2  to  find  a  feasible  solution 
and  for  Phase  3  to  then  obtain  a  final  solution. 


The  next  method  tested  for  generating  multiple  solutions  was 
the  Rx-x-x  procedure  presented  in  Section  2.  Table  VIII  shows  the  results 

of  applying  the  R1-2A-1  version  of  this  new  type  of  procedure  to  the  same 

?9  problems.  Thus,  Solution  0  refers  to  the  ordinary  final  solution 

obtained  by  the  1-2A-1  procedure,  whereas  Solution  i  (i  =  1,  ...  ,  5) 

refers  to  the  solution  generated  by  applying  this  same  procedure  with 
„  .th  (1)  ,  (2) 

..he  i  new  x  and  x  obtained  as  described  in  Section  2. 

The  next-to-iast  column  shows  the  resulting  improvement  over  Solution  0 
by  taking  the  best  of  the  six  solutions.  Comparing  this  column  with  the 
last  column  of  Taole  VI  thereby  provides  one  comparison  of  the  Rx-x-x 
and  x-Rx-x  types  of  procedures.  However,  it  must  be  noted  that  the  specific 
procedure  reported  in  Table  VI  is  l-R^A-l,  where  the  number  of  final  solu¬ 
tions  generated  is  highly  variable  and  possibly  very  large.  Therefore, 
a  better  comparison  night  be  to  the  1-R2A-1  procedure  with  a  =  0,  0.2, 

0.4,  0.6,  0.8,  1.0,  where  six  final  solutions  (at  most)  also  would  be 
generated.  The  last  column  of  Table  VIII  shows  the  improvement  over  the 
ordinary  1-2A-1  procedure  given  by  this  1-R2A-1  procedure. 

The  reason  for  constructing  the  final  column  of  Table  VIII  in  this  way 
is  that  it  essentially  equates  the  computational  effort  for  1-R2A-1  to  that 
required  by  R1-2A-1  to  obtain  the  results  in  the  next-to-last  column.  After 
excluding  the  time  required  to  solve  for  x^  by  the  simplex  method,  the 
time  required  to  obtain  each  new  final  solution  tends  to  be  about  the  same 
whether  it  is  done  by  1-2A-1,  R1-2A-1,  or  1-R2A-1.  Therefore,  the  time 
required  for  each  of  the  last  two  columns  is  approximately  six  times  that 
for  1-2A-1 . 
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TABLE  VIII 


NORMALIZED  DEVIATION  FROM  OPTIMALITY 
FOR  THE  SOLUTIONS  GENERATED  BY  PROCEDURE  R1-2A-1 
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0.216 

0.279 j 

1.044! 

I 

0.321 i 

0.682 

0.845 

0.303 

0.255 

1.603 

II 

0.239 

0.309 

0.069 
;  0.086 

1. 966  0 
0.274  0 
0.440  0 
0.960  0 
0.394 

0.532 
0.516 
0.163 
0.197 
0.081 
0.051 
0 

0.516 
0.131 


0.823 

0.280 

0.484 
0.229 
0. 466 
0.372 
0.044 
0.219 
0.177 
0 

0.018 

0.319 

0 
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'  Improvement 
J  Over  1-2A-1 


! R1-2A-1  |  1-R2A-1 
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I  I 

0.219  j  0.084  j  0 
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Type  &  No. 
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>0.129 
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0 

7 
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0 

0 

0 

0 
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0 

7 

7 

IBM-2 
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0 

0 

0 

5 
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0 
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0 

0 

0 

15 
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;  0.258 
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0.180 
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50 
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* 

, 

* 

* 

* 

* 

* 
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37 
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!  13 

15 

13 

12 

13 

15 

1  1.000 

0 

50 

15 
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0 

1.033 

0.775 

0.516 

0.775 

0 

i 

;  0 

0 

30 

6o 

1-9 

‘>0.465 

>0.272 

>0.578 

>0.468 

>0.597 

>0. 819 

|  0.193 

0.078 
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IBM-7 

>0.625 

>1.148 

>1.082 

- 

j  0.168 

0.042 

; 
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XL.  132 
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>0.633 
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>0.780 | 

l 
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|  >0.264 
! 

>0.270 

>0.378 

>0.283 

>0.310 

0.159 

0.048 

No  feasible  solution  obtained  in  Phase  2. 
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8.2.  Comparison  of  Multiple  Solution  Procedures 

Three  basic  approaches  now  are  available  for  generating  multiple  solutions 
at  one  phase  in  order  to  try  to  improve  upon  the  final  solution  obtained  at  the 
end  of  Phase  3.  One  is  the  x-Rx-x  approach,  which  is  investigated  in  detail 
in  Tables  V,  VI,  and  VII  for  the  x-R3x-l  case.  A  second  is  the  Rx-x-x 
approach,  which  is  studied  in  Table  VIII  for  the  case  of  R1-2A-1.  The  third 
is  to  apply  different  individual  procedures  to  the  same  problem,  e.g.,  by 
varying  the  criterion  used  in  Phase  2. 

One  could  increase  the  number  of  solutions  obtained  even  further  by 
combining  all  three  approaches.  For  example,  consider  all  of  the  pro¬ 
cedures  represented  in  Tables  IV-VIII  as  applied  to  the  same  39  problems. 

On  9  of  these  problems,  the  same  solution  was  obtained  by  essentially 
every  procedure.  However,  there  was  considerable  variation  in  the  solutions 
obtained,  and  in  the  relative  performances  on  the  individual  procedures, 
on  the  other  30  problems.  Specifically,  the  best  solution  v.-as  obtained 
by  the  1-2A-5  procedure  13  times,  by  the  R1-2A-1  procedure  12  times, 
by  some  x-R3x-l  procedure  12  times,  by  the  1-2A-U  procedure  7  times,  and 
by  the  1-2A-3  procedure  7  times.  The  average  normalized  deviation  from 
optimality  of  the  best  solution  obtained  on  each  of  the  39  problems  was 
only  >  0.015,  a  very  substantial  improvement  over  that  obtained  by  any 
one  of  the  procedures.  Furthermore,  this  best  solution  actually  was 
optimal  on  19  of  the  32  problems  where  the  optimal  solution  was  known. 

(Keep  in  mind  that  many  of  these  problems  had  been  selected  as  "difficult" 
ones  for  which  previously  tested  heuristic  procedures  had  not  obtained 
an  optimal  solution. ) 

These  facts  illustrate  a  key  inherent  property  of  heuristic  procedures, 
namely,  the  great  variability  in  their  relative  performances  from  one 
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problem  to  the  next.  Even  though  one  procedure  may  be  distinctly  better 
than  another  on  the  average,  the  inferior  procedure  may  stiUL  obtain  a 
better  solution  on  a  considerable  proportion  of  the  problems.  Given 
the  highly  combinatorial  nature  of  integer  programming  problems,  there 

is  a  good  deal  of  luck  involved  in  uncovering  a  good  (or  optimal)  feasible 
solution,  and  sometimes  a  less  promising  path  will  lead  to  the  prise. 
Therefore,  if  the  only  concern  were  to  maximize  the  quality  of  the  best 
solution  obtained,  then  one  should  indeed  follow  the  old  maxim  to  "try, 
try  again. " 

However,  the  time  and  expense  involved  also  must  be  considered,  of 
course.  One  can  quickly  reach  a  point  of  diminishing  return  where  the 
expected  marginal  improvement  from  additional  runs  becomes  too  small  to 
justify  the  computer  time.  This  is  particularly  true  if  the  new  solutions 
being  obtained  frequently  are  merely  repeats  of  ones  previously  obtained. 
Therefore,  an  important  criterion  for  designing  a  multiple.-solution  pro¬ 
cedure  is  its  tendency  to  generate  new  distinct  solutions. 

Under  this  criterion,  the  approach  of  repeating  the  same  individual 
procedure  with  different  Phase  2  criteria,  as  in  Table  II,  is  not  a  par¬ 
ticularly  good  one.  (Also  see  Table  V.)  Even  varying  the  Phase  1  and  2 
methods  (from  among  the  choices  represented  in  Table  II )  does  not  help 
much.  (Note  that  all  24  procedures  only  obtained  about  2  i  distinct 
solutions  per  problem  in  Table  II.)  Varying  the  Phase  3  method,  as  in 
Table  IV,  does  somewhat  better.  Although  Methods  3  and  4  usually  give 
the  same  solution,  this  solution  frequently  is  different  from  the  Method 
5  solution,  so  running  either  x-x-3  or  x-x-4  along  with  x-x-5  would  be 
reasonable.  Nevertheless,  1-2A-3  and  1-2A-4  each  were  able  to  improve 
upon  1-2A-5  only  five  times  (six  times  combined)  in  Table  IV,  and  the 
improvements  obtained  were  quite  modest. 
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The  x-Rx-x  approach  fares  somewhat  better  under  this  criterion. 

In  fact,  Table  VI  shows  that  1-R3A-1  obtained  an  average  of  U.9  distinct 
solutions  (out  of  10.2  attempts)  per  problem.  Of  the  36  problems  where 

the  first  solution  (the  1-3A-1  solution)  -was  not  optimal,  a  subsequent 
solution  provided  an  improvement  on  17  of  them.  Similarly,  Table  VIII 
indicates  that  1-R2A-1  provided  an  improvement  over  1-2A-1  on  13  of  these 
problems. 

Hovrever,  Table  VIII  demonstrates  that  the  Rx-x-x  approach  is  the  easy 
champion  for  generating  distinct  solutions.  In  fact,  with  the  exception 
of  the  IBM  problems,  practically  every  solution  is  distinct.  Furthermore, 
R1-2A-1  improved  over  1-2A-1  on  23  of  the  problems,  with  an  average  improve¬ 
ment  far  larger  than  either  1-R2A-1  or  1-R3A-1.  Even  when  compared  with 
the  champion  among  the  tested  individual  procedures,  1-2A-5,  R1-'2A-1 
obtained  a  better  solution  on  9  of  the  problems,  and  R1-2A-5  probably  would 
have  done  much  better. 

The  apparent  explanation  for  the  great  success  of  Rx-x-x  in  generating 
distinct  solutions  is  the  diversity  of  its  areas  of  search.  Whereas 
x-Rx-x  continues  searching  along  the  same  line  segment  into  the  feasible 
region,  Rx-x-x  uses  a  completely  different  line  segment  in  a  different 
part  of  the  feasible  region  to  search  for  each  new  solution. 

Since  Rx-x-x  requires  only  about  the  same  computer  time  per  solution 
(and  so  less  per  distinct  solution)  as  the  corresponding  x-Rx-x  procedure, 
it  must  be  rated  as  the  better  of  these  multiple -solution  approaches. 

The  test  results  on  individual  procedures  can  guide  the  choice  of  the 
specific  Rx-x-x  procedure,  e.g.,  R1-3E-5  should  be  a  very  powerful  one. 

However,  the  choice  between  x-Rx-x  and  Rx-x-x  need  not  be  an  either-or 
one.  For  a  fixed  total  number  of  solutions  to  be  generated,  some  can  be 
obtained  in  one  v;ay  and  the  rest  in  the  other.  Since  the  computer  time 


involved  for  obtaining  each  solution  is  about  the  same  for  comparable 
x-Rx-x  and  Rx-x-x  procedures,  this  combined  approach  would  be  advantageous 
if  the  expected  marginal  improvement  from  another  solution  sometimes  would 
be  larger  if  it  is  generated  by  x-Rx-x  rather  than  Rx-x-x.  There  is 
some  evidence  suggesting  that  this  is  indeed  the  case. 

To  begin  examining  this  evidence,  consider  the  "Ph.2"  and  "Ph.2  Sol11" 
columns  of  Table  VI.  Although  there  is  considerable  variation  in  which 
Phase  2  solution  led  to  the  best  final  solution,  a  careful  examination 
suggests  a  definite  bias  toward  the  earlier  (smaller  a)  Phase  2  solutions 
(particularly  when  the  total  number  of  such  solutions  is  relatively  large). 
Compare  the  averages.  Then  note  that  ifl  led  to  the  best  solution  21  out 
of  38  times,  and  #2  did  so  19  out  of  33  possible  times,  whereas  fr4  did 
so  only  8  out  of  2k  possible  times.  Of  21  problems  with  five  or  more 
distinct  Phase  2  solutions,  only  two  required  going  beyond  #4  to  obtain 
the  best  solution.  Ten  problems  required  going  to  #2  to  obtain  the  best 
solution  (contributing  0.058- of  the  0.097  average  decrease  in  the  normalized 
deviation  from  optimality),  and  five  more  required  going  to  either  # 3  or 
#4.  Thus,  going  to  #2  is  very  worthwhile,  but  the  expected  marginal 
improvement  decreases  rapidly  thereafter. 

An  analysis  of  Table  VIII  suggests  that  Rx-x-x  gives  a  slower  decrease 
in  the  expected  marginal  improvement  of  its  successive  solutions  than 
x-Rx-x.  Solution  4  obtained  the  best  solution  as  many  times  (l6)  as 
Solution  1.  Ten  problems  required  going  to  Solution  1  to  obtain  the  best 
solution,  but  13  more  required  going  further  (4  to  Solution  2,  and  3  each 
to  Solutions  3,  4,  and  5).  The  average  marginal  improvement  was  0.097 
for  Solution  1,  0.023  for  Solution  2,  and  a  total  of  0.048  for  the  next 


three  solutions. 


However,  after  going  to  at  least  Solution  1,  the  first  extra  solution 
from  1-R3A-1  (#2)  gave  a  larger  average  marginal  improvement  than  the 
next  R1-3A-1  solution.  After  going  to  at  least  Solution  3,  the  next 
1-R3A-1  solution  (#3)  helped  more  than  the  next  R1-3A-1  solution. 

Another  factor  arguing  for  a  combined  approach  is  that  Rx-x-x  is 
not  well-suited  for  all  problems.  Specifically,  if  the  extreme  points 
adjacent  to  x^  are  located  far  away  from  the  good  feasible  solutions 
(because  their  objective  function  values  are  far  inferior  to  that  for  x^), 
then  Rx-x-x  will  have  little  chance  of  helping.  This  apparently  occurred 
on  problems  1-103,  1-104,  and  la-3  in  Table  VIII,  since  each  of  the  adjacent 
extreme  points  on  these  problems  led  to  final  solutions  with  extremely  large 
normalized  deviations  from  optimality. 

Therefore,  the  recommended  multiple-solution  approach  is  to  combine 
Rx-x-x  and  x-Rx-x.  (denoted  hereafter  as  Rx-Rx-x).  For  a  specified  total 
number  of  solutions  to  be  generated,  the  first  solution  would  be  obtained 
in  the  usual  way,  then  the  desired  number  of  x-Rx-x  solutions,  and  then 
Rx-x-x  would  be  applied  for  the  rest.  (This  idea  could  be  pushed  even 
further  by  using  the  x-Rx-x  approach  to  generate  more  than  one  solution 
along  each  of  the  new  line  segments  generated  by  the  Rx-x-x  approach, 
but  this  would  not  seem  worthwhile  under  ordinary  circumstances.)  To 
be  specific  on  the  allocation  of  effort,  it  is  suggested  that  l/3  of 
the  solutions  (rounded  down)  be  generated  by  x-Rx-x. 

Among  the  alternative  Rx-Rx-x  procedures,  R1-R3E-5  should  be  a  very 
effective  one.  With  x-x-5  rather  than  x-x-1,  the  expected  marginal  improve¬ 
ment  of  successive  solutions  should  decrease  even  more  rapidly,  so  the 
number  generated  should  be  kept  rather  small.  Four  should  be  very  adequate 
for  most  purposes,  and  it  rarely  should  be  worthwhile  to  exceed  ten  or 
fifteen. 
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9.  Comparison  with  the  Ibaraki-Ohashi-Mine  Algorithm 

The  algorithm  presented  by  Ibaraki  et  al  [10]  is  (with  some  modifi¬ 
cations)  a  2-R3-X  procedure,  where  their  Phase  3  method  is  perhaps  most 
similar  to  Method  5  of  the  ones  tested  here.  For  part  of  their  testing 
program,  they  obtained  a  listing  of  all  of  the  author's  test  problems  used 
in  [9]  (except  the  five  "Large  Test  Problems"),  as  given  in  Appendix  1  of  [8]. 
They  report  the  results  of  running  their  algorithm  on  five  of  these  problems, 
namely,  1-5,  1-6,  II— 1,  II-3,  and  II— 11,  achieving  a  normalized  deviation  from 
optimality  of  0.229,  0.018,  0.047,  0,  and  0.012,  respectively,  for  an  average 
of  0.061. 

Compering  these  numbers  with  those  given  in  Tables  IV,  V,  and  VII 
for  these  problems  suggests  that  this  algorithm  is  quite  competitive  with 

1- 2A-5,  x-R5x-l,  or  R1-2A-1.  However,  since  it  is  most  similar  to 

2- Rjx-5>  the  most  interesting  comparisons  would  be  with  this  procedure 
and  its  recommended  variations,  1-R5X-5  and  Rl~R5x-5.  Although  these 
particular  data  are  not  available,  consider  the  approximation  where  1-2A-5 
is  used  for  the  first  solution  and  1-R*x-1  (rather  than  the  mere  powerful 
l-R3x-5)  for  the  rest.  This -approach  obtains  a  better  solution  than  the 
Ibaraki  algorithm  on  four  of  the  five  problems  and  matches  the  optimal 
solution  on  the  fifth.  (When  1-2A-5  is  combined  with  2-R5A-1  or  R1-R2A-1 
(with  few  solutions)  instead,  then  the  one  change  is  that  the  Ibaraki 
algorithm  does  better  on  1-6. ) 

It  must  be  emphasised,  however,  that  more  data  would  be  required  to 
draw  a  conclusive  comparison. 
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Some  of  the  insights  gained  in  this  investigation  nay  have  wider 
applicability  for  the  development  of  heuristic  algorithms  in  various  areas 
of  discrete  programming,  etc.  They  can  at  least  provide  promising  starting 
points  for  guiding  the  development  of  such  algorithms  and  the  resulting 
experience  would  determine  the  extent  to  which  they  are  generally  valid. 
Therefore,  they  are  presented  here  in  the  form  of  hypothesised  principles 
for  heuristic  algorithms.  At  the  same  time,  it  must  be  emphasized  that  these 
are  only  tentative  hypotheses,  based  on  a  limited  set  of  test  problems  which 
may  not  be  representative  of  many  other  discrete  programming  problems. 

Heuristic  can  be  defined  as  "guiding  in  discovery"  or,  in  the  present 
context,  "guiding  in  the  discovery  of  good  solutions."  No  guarantees 
can  be  given  on  how  successful  the  search  actually  will  be,  and  .results 
may  vary  considerably  from  one  attempt  to  the  next.  There  is  a  definite 
element  of  luck  Involved,  where  the  heuristic  procedure  is  designed  to 
weight  the  odds  in  one's  favor  as  much  as  possible.  Therefore,  it  seems 
appropriate  to  use  the  parlance  of  gambling  in  stating  these  hypothesized 
principles.  (Translations  follow  in  parentheses . ) 

Hypothesized  Principle  1:  Keep  your  stakes  high.  (Focus  the  search  where 
any  feasible  solutions  found  should  be  particularly  good  ones,  even  though 
they  are  difficult  to  uncover,  rather  than  cuickly  locating  a  mediocre 
feasible  solution  and  trying  to  improve  upon  it.) 

The  heuristic  procedures  studied  herein  start  the  search  from  the 
"ideal,"  the  linear  programming  solution  x^,  so  any  feasible  solutions 
nearby  will  be  particularly  good  ones  (as  well  as  particularly  difficult 
to  uncover).  The  focus  of  the  search  moves  no  further  away  from  this 
ideal  than  is  necessary  to  find  tne  initial  feasible  solution  for  Fnase  J. 


Early  computational  experience  showed  that  this  approach  was  far  suoerior 

(2) 

to  the  alternative  of  merely  rounding  x'  to  quickly  obtain  a  (mediocre) 
initial  feasible  solution,  and  also  was  markedly  superior  to  such  inter¬ 
mediate  alternatives  as  Method  1  of  Phase  2.  As  discussed  in  Section  6, 
even  the  slight  compromises  involved  in  Method  2  or  Criterion  S  of  Phase  2 
seems  to  have  a  detrimental  effect.  Thus,  there  is  a  rather  strong  positive 
correlation  betv/een  the  quality  of  the  initial  feasible  solution  and  of 
the  final  solution  obtained. 

Hypothesized  Principle  2:  Stick  with  a  winning  streak,  but  don’t  push 
your  luck.  (When  a  good  feasible  solution  is  found,  there  often  is  an 
even  better  one  nearby,  so  continue  searching  in  this  region  until  no 
further  improvement  have  been  obtained  within  a  reasonable  period  of  time.) 

The  first  two  modes  of  search  used  in  most  of  the  Phase  5  methods 
seek  (and  often  find)  small  changes  in  the  current  feasible  solution  that 
yield  improvements.  However,  care  is  taken  not  to  use  a  mode  of  search 
whose  growth  rate  v'ith  problem  size  is  so  large  that  it  can  continue 
fruitlessly  for  an  extended  time.  On  a  broader  basis,  the  x-Rx-x  multiple- 
solution  approach  also  initiates  new  searches  near  where  the  last  successful 
one  began.  As  discussed  in  Section  6,  this  approach  is  a  good  one  if  it 
is  used  in  moderation. 

Hypothesized  Principle  J>i  Occasionally  shuffle  the  deck.  (When  the  search 
has  gotten  too  locked  into  one  well  explored  neighborhood,  then  allow 
it  to  drift  awhile  until  it  homes  in  on  better  solutions  in  an  entirely 
different  part  of  the  feasible  region.) 


Method  5  of  Phase  3  examplifies  the  successful  application  of  this 


principle . 


Hypothesized  Principle  4:  Play  the  field.  (The  search  should  move  into 
several  different  "high  stake"  regions  in  order  to  obtain  a  variety  of 
solutions,  thereby  increasing  the  chances  of  "hitting  the  jackpot”  of 
finding  an  optimal  solution.) 

The  Rx-x-x  multiple -solution  approach  enjoyed  outstanding  success 
in  following  this  principle. 


11.  Summary  of  Recommendations  and  Conclusions 


Although  an  extensive  program  of  computational  experimentation  failed 
to  reveal  significant' differences  between  certain  of  the  leading  methods 
and  criteria,  the  following  heuristic  procedures  can  be  recommended  from 
the  current  evidence  as  at  least  comparable  to  any  of  the  others  on  the 

problems  tested. 

1.  For  a  basic  single-solution  procedure  that  is  both  very  effective 
and  reasonably  efficient,  use  1-3E-5- 

2.  For  a  very  powerful  multiple-solution  procedure,  use  R1-R3E-5 

as  described  at  the  end  of  Section  8.2.  The  total  number  of  solu¬ 
tions  generated  should  be  adjusted  to  fit  the  desired  time-quality 
tradeoff,  but  four  should  be  very  adequate  for  most  purposes. 

3.  For  a  "quick  and  dirty"  procedure  that  is  still  reasonably  effec¬ 
tive,  use  1-2S-3.  (The  procedure  can  even  be  terminated  early 

as  needed  and  still  probably  provide  a  reasonably  good  feasible 
solution  since  it  quickly  finds  an  initial  feasible  solution  and 
then  devotes  the  bulk  of  the  time  to  seeking  progressive  improve¬ 


ments  . ) 
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No  further  comparative  testing  is  planned.  However,  a  modest  amount 


of  additional  developmental  testing  to  refine  these  procedures  appears 
to  be  worthwhile.  The  primary  focus  will  be  on  Method  5  of  Phase  3  to 
see  if  it  can  be  streamlined  greatly  without  significantly  diminishing 
its  effectiveness,  but  several  similar  Questions  about  other  parts  of 
the  procedures  also  will  be  pursued.  Further  information  on  the  growth 
rate  of  execution  time  with  problem  size  also  will  be  sought. 

In  a  parallel  investigation,  Faaland  and  Hillier  [3]  have  analyzed 
the  current  Phase  1  methods  in  detail  and  developed  some  nevT  options. 

Although  the  procedures  tested  here  have  been  presented  in  terms 
of  pure  integer  programming,  they  can  be  extended  quite  readily  to  the 
more  general  case  of  mixed  integer  programming.  Ibaraki  et  al  [10]  have 
done  this  in  one  way.  This  also  has  been  done  in  another  way  by  the 
author,  and  some  preliminary  testing  conducted.  These  results  will  be 
presented  subsequently  in  conjunction  with  the  further  testing  and 
refining  mentioned  above. 

In  many  problems,  the  integer  variables  actually  are  restricted  to 
be  binary  (0-1).  These  procedures  could,  of  course,  be  applied  directly 
to  such  problems  by  merely  introducing  upper  bounds  of  one  on  the  variables 
as  additional  functional  constraints.  However,  this  would  be  a  crude 
and  needlessly  inefficient  way  of  handling  0-1  problems.  What  is  needed 
is  a  thorough  adaptation  and  streamlining  of  the  procedures  and  codes 
to  exploit  tlie  special  structure  of  these  problems.  This  is  being  pursued. 
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APPENDIX 


Further  Comparative  Testing 

In  order  to  more  adequately  compare  the  four  basic  types  of  procedures 
and  the  six  Phase  2  criteria  considered  in  Table  II,  a  supplemental  experimental 
program  was  undertaken. 

The  first  step  was  to  randomly  generate  20  more  Type  I  problems  (labeled 
1-121  to  1-140)  and  20  more^Type  la  problems  (labeled  1-21  to  1-40) ,  each  with 
m  =  10  and  n  =  20.  (These  two  types  were  chosen  since,  based  on  the  resulting 
normalized  deviations  from  optimality,  they  appear  to  be  particularly  difficult 
for  the  heuristic  procedures.)  All  of  the  procedures  and  criteria  were  applied 
to  these  problems,  with  the  results  shown  in  Tables  IX  and  X,  respectively. 

Given  the  analysis  of  computational  results  presented  in  the  early 
part  of  Section  6,  the  next  step  was  to  randomly  generate  200  additional 
problems  of  Type  I  (again  with  m  =  10,  n  =  20 )  to  try  to  distinguish 
between  Criteria  A,  C,  E,  and  S  on  a  sound  statistical  basis.  As  the 
recommended  choice  at  this  point  of  the  four  basic  types  of  procedures,  procedure 
1-5-1  was  used  with  each  of  these  criteria.  On  29  of  the  problems,  the 
simplex  method  found  that  there  was  either  no  feasible  solutions  or  no 
bounded  optimal  solution  x^  for  the  continuous  (linear  programming) 
version  of  the  problem.  Each  of  the  four  criteria  produced  a  feasible 
integer  solution  on  all  of  the  remaining  171  problems.  The  six  pairs 
of  criteria  were  then  compared  on  the  quality  of  their  final  solutions 
on  these  171  problems. 

Let  V^(x)  denote  the  normalized  deviation  from  optimality  of  the 
final  solution  under  Criterion  x  (x  =  A,  C,  E,  s)  on  problem  i 
(i  =  1,  2,  ...  ,  171).  For  a  given  pair  of  criteria,  x  and  y,  let 

Di(x-y)  =  Vi(x)  -  Vi(y),  for  i  =  1,  2,  . . .  ,  171  • 
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TABLE  IX 
(Continued) 


Criterion  Used  in  Phase  2 


Problem 

■B 

C 

D 

i  E 

S 

ESI 

g^l 

0.102 

0.091 

0.204 

0.255 

1-3-1 

0. 074 

:  0. 167 

0.111 

0.131 

0. 148 

0.131 

Ave . 

ggg 

0.291 

! 

0.341 

0.288 

0.280 

0.255 

0.286 

B99 

j  0.101 

1 

0.227 

0.154 

0.238 

0.171 

0.115 

1-2-1 1  0.098 

0.088 

0.074 

0.062 

0.046 

0.099 

1-3-1 

:  0.078 

0.097 

0.083 

0.105 

0.077 

0.138 

w/o 

1-130 

HSifl 

0.134 

0.126 

0.078 

0.132 

BH 

j  0.107 

j  0.126 

.  L.  -  _  _ 

0.076 

0.081 

0.068 

0.121 

1. 

2. 

3- 

4.. 

Value  obtained 
Value  obtained 
Value  obtained 
Value  obtained 

for  1-2-1. 
for  1-3-1. 
for  2-2-1. 
for  2-3-1. 
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TABLE  X 


(Continued) 


J.  Value  obtained  for  2-2-1. 
U.  Value  obtained  for  2-5-1. 


1.  Value  obtained  for  1-2-1 

2.  Value  obtained  for  1-5-1 
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Since  the  problems  are  randomly  generated  (by  the  mixed  congruential  method)  from 
fixed  (discretized  uniform)  probability  distributions,  all  of  these  for  the 

given  x  and  y  are  drawn  from  the  same  underlying  probability  distribution 
(before  conditioning  on  the  parameter  values  taken  on  for  the  particular  problem) . 
In  order  to  distinguish  between  the  two  criteria  involved,  the  objective  is  to 
determine  whether  the  mean  of  this  underlying  distribution  can  be  concluded  to 
be  different  from  zero  and,  if  so,  whether  it  is  positive  (y  better  t  an  x) 
or  negative  (x  better  than  y) . 


Since  any  two  criteria  (x  and  y)  frequently  yield  the  same  final 

solution,  this  underlying  distribution  has  considerable  mass  at  zero, 

which  makes  it  considerably  more  difficult  to  detect  a  nonzero  mean. 

However,  this  mean  indeed  will  be  nonzero  if  the  rest  of  the  distribution 

(the  conditional  distribution  given  nonzero  values)  has  a  nonzero  mean, 

so  it  is  better  to  focus  on  this  conditional  distribution  to  address  the 

questions  at  issue.  Let  p.  denote  the  mean  of  this  conditional  dis- 

x-y 

tribution.  The  nonzero  D^x-y)  values  represent  random  observations 

from  the  conditional  distribution  that  can  be  used  to  obtain  a  point 

estimate  of  p  and  a  confidence  interval  about  p  .  For  this  tyoruose, 
x-y  x-y 

it  is  assumed  that  the  conditional  distribution  is  (approximately)  normal. 

(If  no  assumption  is  made  about  the  functional  form  of  this  distribution, 
the  resulting  confidence  interval  would  be  much  wider,  which  would  further 
reinforce  the  conclusions  stated  after  the  next  paragraph. 

Using  this  approach  for  each  of  the  six  pairs  of  criteria  led  to  the 

results  shown  in  Table  XI.  The  first  column  of  data  shows  the  sample 

size  n  (the  number  of  nonzero  D, (x-y)  values  out  of  the  171  problems), 
x-y  1 

The  next  two  columns  give  the  maximum  likelihood  estimate  of  p  (the 

x-y 

sample  average),  denoted  by  p  ,  and  the  sample  standard  deviation 

x-y 

s  .  The  last  column  presents  the  99j»  confidence  interval  about  p  _ 
x-y  x-y 

(so  that  the  fiduciary  probability  that  all  six  intervals  cover  their 
respective  mc-ans  is  at  least  0. 9*+  by  bhe  Bonferroni  inequality). 
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TABLE  XI 


STATISTICAL  COMPARISON  OF  PHASE  2  CRITERIA 
FROM  200  ADDITIONAL  TYPE  I  PROBLEMS 


x-y 

n 

x-y 

A 

|i 

x-y 

s 

x-y 

99$  Conf.  Int. 

A-C 

41 

+0.013 

0.391 

(-0.152, 

0.178) 

A-E 

42 

+0.025 

0.414 

(-0.148, 

0.198) 

A-S 

69 

-0. o8l 

0.736 

(-0.316, 

0.155) 

C-E 

27 

+0.019 

0.^2 

(-0.164, 

0.202) 

C-S 

73 

-0.084 

0.704 

(-0.302, 

0.135) 

E-S 

74 

-0.089 

0.706 

(-0.307, 

0.128) 

Recalling  that  positive  values  of  favor  y •  over  x, 

it  can  be  seen  that  the  results  in  Table  XI  are  most  favorable  to  Criterion 

E  (by  a  slight  margin  over  C  and  A),  and  least  favorable  to  Criterion  S. 

However,  it  must  be  emphasized  that  none  of  the  confidence  intervals 

exclude  zero,  so  the  null  hypothesis  that  p  =0  cannot  be  rejected 

x-y 

in  any  of  the  six  cases  at  a  99$  level  of  statistical  significance. 

In  fact,  even  for  ‘the  most  extreme  case  of  E-S,  this  hypothesis  cannot 

be  rejected  at  just  the  90$  level,  even  if  the  alternative  hypothesis  is 

the  one-sided  one,  u  _  <  0.  Also  note  how  very  small  the  p  are 

E-S  x-y 

relative  to  the  s  .  Therefore,  the  only  solid  conclusion  that  can 
x-y 

be  drawn  from  these  data  is  that,  even  though  there  will  be  occasional 
large  differences  in  both  directions  on  individual  problems,  any  differences 
in  the  long  run  average  performance  of  these  criteria  should  be  quite  small 
on  problems  similar  to  Type  I. 


